fix(test): create-batch --run without --wait exits 0 on a fully successful dispatch#267
Conversation
…ssful dispatch
The no-wait fan-out returns each result with the trigger response's status,
which is `queued` by design (non-terminal — the field is documented as
"Terminal status if --wait; queued if no --wait"). But the exit-code logic
only recognized terminal statuses:
const allPassed = batchRunResults.every(r => r.status === 'passed');
Every no-wait result is `queued`, so `allPassed` was always false and the
command fell through to exit 1 with a misleading "N of N run(s) did not
pass" — even when every trigger was dispatched successfully. This broke the
documented exit-code contract (the DOCUMENTATION.md create-batch example
uses `--run --max-concurrency 4 --output json`, no `--wait`), failed CI on
fully successful dispatches, and was internally inconsistent with single
`test run <id>` without `--wait`, which exits 0 on a successful `queued`
dispatch.
In the no-wait case "success" means every trigger dispatched without error
(statuses are non-terminal by definition). The exit-code block now branches
on `opts.wait`:
const failing = opts.wait
? batchRunResults.filter(r => r.status !== 'passed')
: batchRunResults.filter(r => r.error !== undefined);
if (failing.length === 0) return; // exit 0
Trigger errors in no-wait mode keep today's aggregation (uniform specific
code when shared, else exit 1), and the "did not pass" message becomes
"N of N trigger(s) failed" when `--wait` is absent. The stderr text summary
is likewise corrected for no-wait mode — it now reports "N/N triggered"
instead of misreading a successful dispatch as "0/N passed".
Adds three specs in test.create-batch-run.spec.ts: all-queued (json) →
resolves without error and never polls; all-queued (text) → "N/N triggered"
summary, no "passed"/"did not pass" wording; partial trigger failure → still
exits non-zero with the full results envelope. The two success specs fail
against the pre-fix exit-code block and pass with it. Existing no-wait specs
only exercised error scenarios and never asserted the success exit code,
which is how this slipped through.
Fixes TestSprite#161
fe26b4a to
8f463b3
Compare
WalkthroughChangesBatch run no-wait behavior
Estimated code review effort: 3 (Moderate) | ~20 minutes Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Warning There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure. 🔧 ESLint
ESLint install timed out. The project may have too many dependencies for the sandbox. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/commands/test.ts`:
- Around line 2938-2942: Update the no-wait error branches around the allTimeout
and corresponding failure handling to use trigger/dispatch terminology instead
of batch run terminology, including the messages and any related counts or
descriptions. Preserve the existing behavior and leave the default wait-path
wording unchanged.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: d507dbbb-ff90-47dd-977e-f2e909389ebb
📒 Files selected for processing (3)
DOCUMENTATION.mdsrc/commands/test.create-batch-run.spec.tssrc/commands/test.ts
| if (allTimeout) { | ||
| throw new CLIError( | ||
| `All ${batchRunResults.length} batch run(s) timed out after ${timeoutSeconds}s.`, | ||
| 7, | ||
| ); |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Keep no-wait failures in trigger terminology.
These branches can run with wait: false, but report timed-out/failed runs. No-wait only dispatches triggers; the default branch at Line 2963 already uses the correct terminology.
Proposed fix
- `All ${batchRunResults.length} batch run(s) timed out after ${timeoutSeconds}s.`
+ opts.wait
+ ? `All ${batchRunResults.length} batch run(s) timed out after ${timeoutSeconds}s.`
+ : `All ${batchRunResults.length} trigger(s) timed out after ${timeoutSeconds}s.`
- `Batch run finished: ${failing.length} run(s) failed with exit code ${uniformCode}.`
+ opts.wait
+ ? `Batch run finished: ${failing.length} run(s) failed with exit code ${uniformCode}.`
+ : `Batch run trigger finished: ${failing.length} trigger(s) failed with exit code ${uniformCode}.`As per path instructions, text should match the underlying dispatch/trigger outcomes.
Also applies to: 2953-2957
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/commands/test.ts` around lines 2938 - 2942, Update the no-wait error
branches around the allTimeout and corresponding failure handling to use
trigger/dispatch terminology instead of batch run terminology, including the
messages and any related counts or descriptions. Preserve the existing behavior
and leave the default wait-path wording unchanged.
Source: Path instructions
|
Right shape for #161: the exit-code tail now branches on --wait — no-wait success = every trigger dispatched (a queued result counts) → exit 0, the --wait path stays behavior-preserving, and the documented batch semantics hold (all-conflict → 6, deferred/timeout → 7, empty-batch safe); partial no-wait failures now surface their specific error code rather than a generic 1. 3 tests covering each behavior. Thanks @kshitij-heizen! |
What does this PR do?
Fixes
test create-batch --runwithout--waitalways exiting 1 even when every trigger succeeds — which broke the documented exit-code contract and failed CI pipelines on fully successful dispatches.The no-wait path returns each result with the trigger response's status, which is
queuedby design (the field is documented as "Terminal status if--wait;queuedif no--wait"). But the exit-code logic only recognized terminal statuses:Every no-wait result is
queued, soallPassedwas always false and the command fell through to exit 1 with a misleading "N of N run(s) did not pass" — even on a fully successful dispatch. It was also internally inconsistent: singletest run <id>without--waitexits 0 on a successfulqueueddispatch.Fix
In the no-wait case, "success" means every trigger was dispatched without error (statuses are non-terminal by definition). The exit-code block now branches on
opts.wait:Trigger errors in no-wait mode keep today's aggregation (uniform specific code when shared, else exit 1), and the "did not pass" message becomes "N of N trigger(s) failed" when
--waitis absent. The stderr text summary is likewise corrected for no-wait mode — it now reportsN/N triggeredinstead of misreading a successful dispatch as0/N passed.--waitsemantics are unchanged.Reproduction (before this PR)
Related issue
Fixes #161
Type of change
Checklist
mainbranch.npm run lintandnpm run format:checkpass.npm run typecheckpasses.npm testpasses and coverage stays above the 80% gate (1618 tests; 88.34% stmts / 85.42% branch).DOCUMENTATION.md(create-batch now documents the no-wait exit-0 contract, mirroringtest run).Notes for reviewers
test.create-batch-run.spec.ts:--output json) → resolves without error and never polls (pollCount === 0);--output text) → summary reads3/3 triggered, with nopassed/did not passwording;NOT_FOUND) → still exits non-zero, full results envelope intact.runBatchRun; JSON output shape,--waitsemantics, and all other exit codes (6/7/11) are unchanged.Summary by CodeRabbit
Bug Fixes
test create-batch --runbehavior when used without--wait.Documentation